Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Idefics3 - resolve data_format for idefics3 image processing #33599

Draft
wants to merge 50 commits into
base: main
Choose a base branch
from
Draft

[DO NOT MERGE] Idefics3 - resolve data_format for idefics3 image processing #33599

wants to merge 50 commits into from
+4,497 −1

Conversation

amyeroberts
Copy link
Collaborator

Demo of changes for image processing to set data_format and input_data_format in idefics 3's image processing

andimarafioti and others added 30 commits September 18, 2024 12:30
@andimarafioti
Add Idefics 3!
842a28d
@andimarafioti
fixes to make both pipelines identical
afce007
@andimarafioti
fix for quantized models
3e3b31d
@andimarafioti
First pass at the review
9c8ffc4
@andimarafioti
remove vocab size from the main config (it's still in the text_config)
7e3d7a6
@andimarafioti
hot fix for merve
dd99bca
@andimarafioti @amyeroberts
Apply suggestions from code review
ddac9ec 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti
re-add model_type for text_config
188bb76
@andimarafioti
remove support for old_cache
43fb214
@andimarafioti
remove hidden_size from main config
c9e0d85
@andimarafioti
rename idefics3 HF repo
1b2b89c
@andimarafioti
few changes suggested in the PR
6ff766f
@andimarafioti
fix to input_data_format computation
11c2e1a
@andimarafioti
remove overwrite of _autoset_attn_implementation following @zucchini-nlp
c1048ed 
 suggestion
@andimarafioti
improve example
a163564
@andimarafioti
few improvements from amy's review
6f0a479
@andimarafioti
big change to enable processing input images as numpy arrays
8361fce
@andimarafioti
Changes to the code to uniformize processor kwargs
32970d0
@andimarafioti
image processing tests
c504f00
@andimarafioti
image processing tests fixes and some bugs they discovered
a914e41
@andimarafioti
addressed review comments from Yoni
6722d13
@andimarafioti
fix modeling tests
0533eda
@andimarafioti
remove special tokens that are not special
b034091
@andimarafioti
fixes tests
47fb7ce
@andimarafioti
skip failing tests - they also fail for idefics2
4032a6f
@andimarafioti
added paper and readded the tests with multi gpu, who knows
757e834
@andimarafioti @amyeroberts
Update docs/source/en/model_doc/idefics3.md
7797279 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti @amyeroberts
Apply suggestions from code review
b478124 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti
review amy until image_processing_idefics3
ada6219
@andimarafioti
last comments from Amy
164fbe8
andimarafioti and others added 20 commits September 18, 2024 12:31
@andimarafioti
review amy
000c8ea
@andimarafioti @amyeroberts
Update src/transformers/models/idefics3/image_processing_idefics3.py
4d02e0c 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti @amyeroberts
Update src/transformers/models/idefics3/modeling_idefics3.py
3bf03c2 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti @amyeroberts
Update docs/source/en/model_doc/idefics3.md
57bfd51 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti
doc improvement - amy review
63b1d7f
@andimarafioti
fix runtime error during fine-tuning
6325fbc
@andimarafioti
amy's review
76b8892
@andimarafioti @amyeroberts
Update src/transformers/models/idefics3/image_processing_idefics3.py
9a20306 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti @amyeroberts
Update src/transformers/models/idefics3/image_processing_idefics3.py
3129920 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti @amyeroberts
Update src/transformers/models/idefics3/modeling_idefics3.py
e1a10b3 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti
ruff
4c3756f
@andimarafioti
amy's comment on the order
fbaf07e
@andimarafioti
ruff ruff
87fa179
@andimarafioti
fix copies
23d4cf8
@andimarafioti
square images when they are not splitted
9e925b9
@andimarafioti
ruff :(
215b636
@andimarafioti @amyeroberts
Update src/transformers/models/idefics3/image_processing_idefics3.py
2967974 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti @amyeroberts
Update tests/models/idefics3/test_processing_idefics3.py
ee041bf 
Co-authored-by: amyeroberts <[email protected]>
@andimarafioti
fix small bug introduced in refactor
4aad266
@amyeroberts
Idefics3 - resolve data_format for image processing
ab8e6dd
@amyeroberts amyeroberts changed the title [DO NOT MERGE] Idefics3 - resolve data_format for [DO NOT MERGE] Idefics3 - resolve data_format for idefics3 image processing Sep 19, 2024
@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@amyeroberts @HuggingFaceDocBuilderDev @andimarafioti